[Evaluation] Enable ToolCallAccuracy/Input/Success on restricted-tool conversations#47369
Open
mmkawale wants to merge 4 commits into
Open
[Evaluation] Enable ToolCallAccuracy/Input/Success on restricted-tool conversations#47369mmkawale wants to merge 4 commits into
mmkawale wants to merge 4 commits into
Conversation
These three evaluators grade the agent's tool selection, input arguments,
and call status -- none consume the (redacted) tool output body -- so the
previous unconditional rejection of conversations containing built-in
restricted tools (bing_grounding, bing_custom_search, azure_ai_search,
azure_fabric, sharepoint_grounding) is now lifted.
Implementation:
- Set check_for_unsupported_tools=False on each evaluator's input validator
in _tool_call_accuracy.py, _tool_input_accuracy.py, _tool_call_success.py.
- The underlying ToolDefinitionsValidator / ToolCallsValidator classes are
unchanged; GroundednessEvaluator and ToolOutputUtilizationEvaluator still
reject restricted tools because they require the tool output body.
Tests:
- New test_unsupported_tools_validation.py (26 tests) covers:
* 15 parametrized cases: each of the 3 evaluators x 5 restricted tools,
asserting validate_eval_input returns True for response= payloads.
* 1 mixed-tools case.
* 10 regression cases asserting the underlying validators still reject
restricted tools when check_for_unsupported_tools=True.
Versioning:
- Bumped _version.py 1.17.0 -> 1.17.1.
- Added 1.17.1 (Unreleased) section to CHANGELOG.md under Features Added.
github-actions
Bot
added
Community Contribution
Contributor
|
Thank you for your contribution @mmkawale! We will review the pull request and get back to you soon. |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Updates tool-related evaluators to allow conversations that include restricted built-in tools by disabling unsupported-tool checks in their input validators, and adds regression tests to ensure the relaxed behavior is limited to those evaluators.
Changes:
- Set
check_for_unsupported_tools=FalseforToolCallAccuracyEvaluator,_ToolInputAccuracyEvaluator, and_ToolCallSuccessEvaluatorvalidators. - Added unit tests covering acceptance of restricted tools for those evaluators and continued rejection when validator flags are enabled.
- Bumped package version and documented the behavior change in the changelog.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_accuracy/_tool_call_accuracy.py | Disables unsupported-tool checking in ToolCallsValidator wiring. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_input_accuracy/_tool_input_accuracy.py | Disables unsupported-tool checking in ToolDefinitionsValidator wiring. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_tool_call_success/_tool_call_success.py | Disables unsupported-tool checking in ToolDefinitionsValidator wiring. |
| sdk/evaluation/azure-ai-evaluation/tests/unittests/test_unsupported_tools_validation.py | Adds regression tests ensuring restricted tools are accepted only where intended. |
| sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_version.py | Bumps version to 1.17.1. |
| sdk/evaluation/azure-ai-evaluation/CHANGELOG.md | Documents the new behavior under 1.17.1 (Unreleased). |
Comment on lines
+83
to
+84
| # Should not raise EvaluationException; flag flip made this path legal. | ||
| assert evaluator._validator.validate_eval_input(eval_input) is True |
Comment on lines
+70
to
+73
| @pytest.mark.usefixtures("mock_model_config") | ||
| @pytest.mark.unittest | ||
| class TestRestrictedToolValidationLifted: | ||
| """Validator should no longer reject restricted tools for these three evaluators.""" |
Comment on lines
+59
to
+67
| def _restricted_tool_definition(tool_name: str): | ||
| return { | ||
| "name": tool_name, | ||
| "description": f"Built-in {tool_name} tool.", | ||
| "parameters": { | ||
| "type": "object", | ||
| "properties": {"query": {"type": "string"}}, | ||
| }, | ||
| } |
Comment on lines
+34
to
+40
| RESTRICTED_TOOL_NAMES = [ | ||
| "bing_grounding", | ||
| "bing_custom_search", | ||
| "azure_ai_search", | ||
| "azure_fabric", | ||
| "sharepoint_grounding", | ||
| ] |
When any tool_call or tool_result in the response carries a known-failure status (failed, error, incomplete, cancelled/canceled), short-circuit _do_eval to return a deterministic fail result (score=0, _passed=False, _result='fail') without invoking the LLM. The evaluator's scoring contract is explicitly binary -- 'FALSE: at least one tool call failed' -- and the prompty rubric does not consider the status field, so it would otherwise grade only the (typically empty) result body and frequently mis-score failed conversations as passes. Reuses the existing pre-flow short-circuit pattern (_is_intermediate_response / _return_not_applicable_result) for consistency. Status is only populated by upstream converters that preserve it; absent status, behavior is unchanged. Bumps to 1.17.1, adds CHANGELOG entry, and adds 19 focused unit tests.
… namespace Brings _ToolInputAccuracyEvaluator in line with its three sibling tool evaluators (ToolCallAccuracyEvaluator, _ToolCallSuccessEvaluator, _ToolOutputUtilizationEvaluator) which are already exposed on the top-level package. Consumers (notably the Foundry evaluations service catalog) can now import it from azure.ai.evaluation directly instead of reaching into the private _evaluators._tool_input_accuracy submodule.
mmkawale
pushed a commit
to mmkawale/azure-sdk-for-python
that referenced
this pull request
Jun 8, 2026
…ery/input fallback for AIS, SP, Fabric break_tool_call_into_messages previously had no elif branch for bing_custom_search or sharepoint_grounding, so calls touching either tool were silently dropped before any evaluator could see them. The three status-only tool evaluators (ToolCallAccuracyEvaluator, _ToolInputAccuracyEvaluator, _ToolCallSuccessEvaluator) therefore returned NOT_APPLICABLE on those conversations even after the validator was loosened in PR Azure#47369. Changes: - bing_custom_search: arguments-only branch mirroring bing_grounding (emits a tool_call with the requesturl; no tool_result, since Bing-family results are redacted upstream for compliance). - sharepoint_grounding: arguments + dumped output, mirroring azure_ai_search. Phase 2 will extend the Groundedness extractor to walk the documents structure already present on the tool_result. - azure_ai_search, sharepoint_grounding, fabric_dataagent input branches: switched from direct details[<tool>][input] dereference to .get(input) or .get(query) or empty-string fallback. Live agent traces emit the search term under 'query' for all three, which made the existing AIS and Fabric branches surface empty arguments to evaluators (a live bug, not just a Phase 1 prerequisite). - Refreshed the stale March-2025 top-of-function comment to reflect the current set of supported built-ins. Tests: Added 5 new tests in tests/converters/ai_agent_converter/test_ai_agent_converter_internals.py covering bing_custom_search, sharepoint_grounding (input key and output dump), and the query-key fallback for AIS, SP, and Fabric. The new tests construct ToolCall via a small _HybridDict helper instead of going through ToolDecoder, so they do not depend on the agents SDK RunStep* models that have moved between azure.ai.projects.models and azure.ai.agents.models packages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable
ToolCallAccuracyEvaluator,_ToolInputAccuracyEvaluator, and_ToolCallSuccessEvaluatoron conversations that contain built-in restricted tools (bing_grounding,bing_custom_search,azure_ai_search,azure_fabric,sharepoint_grounding).These three evaluators grade the agent's tool selection, input arguments, and call status — none of them consume the (redacted) tool output body — so the previous blanket rejection was overly conservative.
GroundednessEvaluatorandToolOutputUtilizationEvaluatorstill reject restricted tools because they require the tool output body.Implementation
check_for_unsupported_tools=Falseon each evaluator's input validator in:_tool_call_accuracy.py_tool_input_accuracy.py_tool_call_success.pyToolDefinitionsValidator/ToolCallsValidatorclasses are unchanged; restricted-tool rejection remains the default for any other caller.Tests
New
test_unsupported_tools_validation.py(26 tests, all passing):validate_eval_inputreturnsTrueforresponse=payloads.check_for_unsupported_tools=True(default for other callers).Versioning
_version.py1.17.0 -> 1.17.1.1.17.1 (Unreleased)section toCHANGELOG.mdunder Features Added.Related
Parallel change in
azureml-assetsregistry (separate PR) bumps the published evaluator versions so the ACA / Foundry batch path picks up the same behavior.